Adjective Scale Probe: Can Language Models Encode Formal Semantics Information?

It is an open question what semantic representations transformer-based language models can encode and whether they have access to more abstract aspects of meaning. Here, we propose a diagnostic dataset investigate how well understand the degree semantics adjectives. In dataset, referred as Adjective Scale Probe (ASP), semi-automatically generate 8 tests Natural Language Inference (NLI) questions test key capabilities adjective interpretation. We apply ASP evaluate performance 3 models, i.e., BERT, DeBERTa, T0. found that perform below majority baseline for most ASP, even when been fine-tuned achieve high on large-scale MNLI dataset. But after fine-tune pre-trained subset DeBERTa untrained adjectives tests, suggesting may captured information through pre-training but it needs specific training data learn such current tasks. sum, provides easy-to-use method fine-grained formal properties adjectives, reveals models' abilities information.